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ABSTRACT 

The impact of teacher observations in alignment with professional development (PD) on teacher 
efficacy was quantified for 292 schools in 110 districts within 27 U.S. States. Teacher 
observations conducted by school leaders or designated internal coaches were coordinated with 
PD offerings aligned with intended teacher improvements. The PD involved throughout was an 
online, on-demand system teachers accessed as convenient with a range of PD assistance 
regarding teaching techniques and participative teacher/user interactive communities for 
collaboratively posting and downloading PD-related materials. Results indicate that systemic 
teacher observations, coupled with aligned PD, resulted in significantly improved student 
achievement in reading and math on standardized assessments. 
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INTRODUCTION 

major purpose of schools and the school experience - arguably the major purpose - is to provide 
teacher-student settings and interactions that result in improved student capabilities and 
achievements. Schools and school districts struggle to identify ways to continuously improve the 
impact teachers have on their students (c.f. Duke & Stiggins, 1990; U.S. Department of Education, 2013). 
Professional development and teacher observation are two example approaches school leaders invest in to ensure 
that teachers have a positive impact on student achievement. 

Professional development (PD) remains a widespread approach aimed at improving teacher efficacy and 
impacts on students (Borko et al., 2006; Buczynski & Hansen, 2010; Fishman et al., 2003; Loucks-Horsley et ah, 
2003; Mizell, 2010; Shaha et al., 2004). 

Despite increased interest and expectations regarding PD, data substantiating improved impact of teachers 
on students remains sparse (Avalos, 2011; Desimone et al., 2002; Farnsworth et al., 2002; Garet et al., 2001; Lewis 
et al., 2003; Shaha et al., 2004). Such is especially the case for traditionally formatted PD approaches requiring 
teacher absences from classroom settings for participation and the expense and burdensome requirements to staff 
replacements. 

The advent of Internet and expanding Internet video now provides PD for educators with fully convenient 
access and a more cost-effective approach to PD than traditional conference approaches requiring teacher absences 
from the classroom (Boling, 2007; Colestock & Sherin, 2009; Sherin & van Es, 2009). A growing number of recent 
rigorous studies have verified that teacher participation in Internet-based, online PD approaches is associated with 
favorable impacts on student performance (Buczynski & Hansen, 2010; Desimone et al., 2002; Magidin et al., 2012; 
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Masters et al., 2012; Rienties et al., 2013; Shaha & Ellsworth, 2013, 2014; Wasik & Hindman, 2011). Advanced 
Internet-based PD offerings specifically do not only include teacher-selected instructive videos and content, but also 
teacher communities within which PD participants can post and download PD-related materials from and for each 
other (c.f. Shaha & Ellsworth, 2013, 2014). 

In summary, findings from studies of online, on-demand, Internet-based PD have verified quantifiable, 
beneficial impacts on student performance. Arguably, online, on-demand PD provides a more cost-effective 
favorable approach than alternatives that require teacher absences and replacement. Thus, educators making 
decisions regarding investments in PD can now do so more enlightened based on research rather than based on 
teacher testimonials and marketing materials alone as was tradition. 

Teacher Observations is another technique widely used wherein school leaders, experts, or designated 
coaches watch teachers and then provide feedback or guidance aimed at improving impact for classrooms and 
student learning. The school reform efforts of the 1980s and 1990s increased attention on observation-based teacher 
evaluations as a critical lever for improving the teaching quality and impact (c.f. Brandt, 1995; Darling-Hammond, 
1990). Observation-based teacher assessments have continued to be predominately structured as once or twice 
yearly occasions on which administrators complete a checklist and ratings forms (Peterson, 2004; Weisberg, et al., 
2009). 


Teacher evaluation criteria has evolved over the past decades, as have beliefs and values surrounding the 
role of teachers, effective teaching, and theories of student learning (Cuban, 1993; Daley, & Kim, 2010; Darling- 
Hammond et al., 1983; Ellett, 1997; Ellett & Teddlie, 2003). Observation-based assessment of teacher efficacy and 
quality have concurrently evolved from ethical perspectives reflecting teacher personal traits toward observable 
behaviors and practices (Ellett & Teddlie, 2003; Good & Mulryan, 1990). 

Yet a review of the literature shows that typical teacher evaluations conducted by administrator 
observations have not proved consistently effective beyond meeting bureaucratic requirements (Darling-Hammond, 
1986; Hazi & Arredondo Rucinski, 2009; Jacob & Lefgren, 2008; Peterson, 2000; Stiggins & Bridgeford, 1985; 
Weisberg et al., 2009). Critiques include inadequate teacher training to remedy observed areas for improvement and 
weak proof that impacts of observations result in any level of higher student achievement (Darling-Hammond, 1986; 
Haefele, 1993; Jacob & Lefgren, 2008; Medley & Coker, 1987; Peterson, 2000). As a result, observations have 
suffered from mixed impacts or even minimal effectiveness. 

In summary, teacher observations came with the goal that first-hand educator assessments would lead to 
identification of teacher strengths and weaknesses, followed by improved teacher capabilities and performance 
through improved teaching. Some studies have appropriately shown observations to be positive. However, to 
maximize effectiveness, it seems that observations and feedback to teachers should be focused on helping identify 
areas for their continuous improvement, resulting in increased impactfulness on students (Ellett & Teddlie, 2003; 
Jacob & Lefgren, 2008; Weisberg et al., 2009). 

SUMMARY 

What is needed is evidence that a coordinated use of improvement-focused teacher-observations with skill¬ 
enhancing, readily accessible and adaptable PD can favorably impact teacher efficacy, as quantified in increases in 
student performance. What is missing is the tight alignment of observed needs with online PD offerings, the latter 
designed to help teachers improve, where needed, as measured in student improvements and achievement. The 
question is whether observation-based identification of teacher needs can lead to highly impactful PD 
improvements. 

Studies continue to stress the ultimate goal of observations and PD focused on improved impacts (Gordon, 
Kane, & Staiger, 2006; Rivkin et al., 2001; Rockoff, 2004; Rowan et al., 2002; Shaha et al., 2004; Wright, Horn, & 
Sanders, 1997). To date, little quantitative evidence remains missing which shows that observation-based feedback 
can be combined effectively with PD to improve teacher skills and capabilities and their impact on student 
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performance. Thus the purpose of the study described herein, which is long needed and will prove impactful for all 
educators focused on continuous teacher improvement for achieving improved student performance. 

METHOD 

Data analyzed included 292 schools in 110 districts within 27 states, representing an estimated 187,000 
students within 54% American states. School inclusion required a minimum of 15 teachers per school participating 
in the same commercially-available online, on-demand PD product widely used in the United States (PD 360 ® and 
Observation 360 ®, aka Edivation ®, School Improvement Network, Salt Lake City). This PD enabled teachers to 
participate in a full range of activities that ranged from instructional videos and content on specific teaching 
techniques and classroom challenges to user communities enabling posting, downloading and exchange of PD- 
related materials. 

Analyses of student performance across varying state locations and their respective assessments required a 
basis for collectively interpretable data. For this reason, status in student performance was quantified based upon 
the percentage of students classified as Proficient or Advanced in Math and Reading/Language Arts collectively by 
school. Within the 27 U.S. states included in the data, levels of student achievement were already categorized on 
scales within which students performing above minimum or average grade-defined levels were classified as either 
Proficient or Advanced in skill level, the second being the higher attainment. Using the sum of the percentage of 
students classified as either Proficient or Advanced provided the basis for collective analyses across settings and 
assessment instruments, as previously validated through published research. Data labelled as “Pre” reflected the 
2010-2011 school year, while “Post” data were for the 2011-2012 school year. 

Additional variables beyond student performance were collected reflecting levels of PD participation 
collectively by schools. Participation data were extracted from those automatically captured by the PD provider as 
an automated, automatic result of PD use. Data, summarized by school, reflected the following variables for 
analysis: 


• Mean number of observations per teacher 

• Means observations with assignments per school (i.e., across teachers) 

• Average minutes viewed (i.e., PD video time) 

• Forums viewed per teacher 

• Forums posted per teacher 

• Files downloaded per teacher 

• Files uploaded per teacher 

• Links downloaded per teacher 

• Links uploaded per teacher 

Thus, the first data element - Mean number of observations per teacher - was captured as recorded within 
the PD application. This reflected the intentional connectivity between the numbers of observer-reported 
observation attempts as recorded within the online PD solution. The purpose of these observation events was, by 
instruction to observers, to determine teacher strengths and limitations and then to recommend the PD they 
subsequently should undertake for capability and impact enhancement purposes. 

Schools (i.e., 292) were classified based upon the mean number of observations per teacher - ties were not 
an analytic challenge. Classifications were then labelled as Higher versus Lower Observation Rate groups (i.e., 
halves) which were created by dividing the schools into two equal groups of 146 each. The resulting mean number 
of teacher observations was 8.49 for the schools classified as Higher Observation Rate versus 2.76 for Lower 
Observation Rate schools. 

Analyses followed a usual quasi-experimental design (Cook & Campbell, 1979). All analyses were 
conducted using SPSS version 17.0 or higher (PASW Statistics, SPSS, 2009) and Excel (Microsoft Office 2007). 
Minimum level of statistical significance was determined a priori at p<0.05. 
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The data amassed reflected the combined effects of teacher observations and PD wherein the use of PD was 
often or always focused on observed areas of teacher need matched with appropriate PD usage. 

Math 


Schools with Higher Observation Rates experienced significantly greater gains in student performance for 
Math (see Table 1). Interestingly, and not by design, schools with the Higher Observation Rates had Pre-PD student 
performance levels 7% lower than their Lower Observation Rates counterparts. However, the difference reversed or 
"flip-flopped” by the end of the PD year wherein the Higher Observation Rates schools shared an 11.5% advantage 
versus their Lower Observation Rates counterparts (see Figure 1). 


Table 1: Comparative Student Gains (Math) 


Math Pet. Advanced + Proficient 


Pre 

Post 

Net Change 

Pet. Change 

p-Value 

LOWER Observation Rates 

56.92 

59.14 

2.22 

3.9% 

p=0.016 

HIGHER Observation Rates 

52.93 

66.10 

13.16 

24.9% 

p=<0.001 

Significant Average 

p<0.001 

pcO.OOl 

p<0.001 

pcO.OOl 




Net Change in Math Performance 



SLOWER Obs Rate 
^—HIGHER Obs Rate 


Pre 


Post 


Figure 1: Comparative Improvement In Student Performance For Math 


Schools with Lower Observation Rates collectively experienced 2.22 net gain in percent of students rated 
as Proficient or Advanced (see Table 1), which represented a statistically significant 3.9% improvement (p=0.016). 
In contrast, schools with Higher Observation Rates collectively experienced 13.16 net gain in the percentage of 
students classified as Proficient or Advanced, representing a statistically significant 24.9% improvement from 
baseline (p<0.001). The comparison reflects 6.37 times the growth in the Higher Observation Rate schools 
compared with the Lower Observation counterparts. 


As anticipated, teachers in Higher Observation Rates schools also experienced significantly higher 
utilization and engagement within the PD on several of the additional variables measured (see Table 2). This is 
interpreted as a reflection of the impact of the observations. Teachers in the Higher Observation Rates schools had 
31.8% more observations with assignments, 39.5% more minutes of PD viewing, and 5.1% more minutes of viewing 
per target (i.e., module). Thus, those more observed also used the PD more plentifully. 
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Table 2: Comparative Teacher Engagement With Observation 



Mean Obs w/Assignments 

Ave. Minutes Viewed 

Minutes Viewedper Target 

LOWER Observation Rates 

13.56 

55.13 

15.49 

HIGHER Observation Rate 

17.87 

76.88 

16.28 

Significant Average 

p=0.048 

p=0.032 

p=0.050 


Reading 


Schools with Higher Observation Rates also experienced significantly greater gains in student performance 
in Reading (see Table 3). Again interestingly, and still not by design, schools with the Higher Observation Rates 
had Pre-student performance levels 12% lower than their Lower Observation Rates counterparts. By the end of the 
PD year, the Higher Observation Rates schools nearly closed the massive pre-existing gap or disadvantage (see 
Figure 2). 


Table 3: Comparative Student Gains (Reading) 


Reading Pet. Advanced + Proficient 


Pre 

Post 

Net Change 

Pet. Change 

p=Value 

LOWER Observation Rates 

62.68 

64.89 

2.21 

3.5% 

p=0.016 

HIGHER Observation Rates 

50.60 

63.07 

12.47 

24.6% 

p=0.001 

Significant Average 

p=0.001 

ns 

p<0.001 

p<0.001 




Net Change in Reading Performance 



- LOWER Obs Rate 

- HIGHER Obs Rate 


Pre 


Post 


Figure 2: Comparative Improvement In Student Performance For Reading 


Schools with Lower Observation Rates collectively experienced 2.21 net gain in the percent of students 
rated as Proficient or Advanced, which represented a statistically significant 3.5% improvement (p=0.016). In 
contrast, schools with Higher Observation Rates collectively experienced 12.47 net gain in student performance for 
percent Proficient or Advanced, representing a statistically significant 24.6% improvement (p<0.001) over baseline. 
The comparison reflects 7.01 times the growth in the Higher Observation Rate schools than was found for the Lower 
Observation counterparts. 


Again as anticipated, data verified that teachers in Higher Observation Rates schools also used the PD at 
significantly higher rates of utilization and engagement within several of the additional variables measured (see 
Table 4). As with Math data, this is interpreted as a reflection of the impact of the observations as guidance and 
motivation to engage in the PD for improvement. Teachers in the Higher Observation Rates schools had more 
(3.2%) observations with assignments, 29.9% more minutes of PD viewed, and 30.1% more files downloaded. 
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Table 4: Further Comparative Teacher Engagement With Observation 



Percent Users w/ Viewing 

Ave. Minutes Viewed 

Files Downloaded 

LOWER Observation Rates 

0.79 

62.17 

24.42 

HIGHER Observation Rates 

0.82 

80.47 

31.79 

Significant 

p=0.038 

p=0.012 

p=0.010 


CONCLUSIONS & RECOMMENDATIONS 

Review of the literature revealed that efforts to maximize the impact of teachers on their students through 
teacher observations suffered generally from minimal effectiveness. Research proving the impact of professional 
development, however, has matured toward proof of impactfulness verified through significant improvement 
changes in student performance. The question remained as to the impact of the combination of the two approaches, 
with the anticipation that each and both would be more impactful together. 

Data showed that students of teachers observed more frequently, as part of a coordinated undertaking and 
in conjunction with accessibility of an online, on-demand PD offering, collectively had greater gains in the 
performance of their students than those who had fewer observations. These same higher-observation teachers also 
proved to use the PD product more often and more actively, reflecting appropriate propensity among observers to 
recommend or direct specific PD undertakings in conjunction with specific observed areas for teacher improvement. 
The design of the study indicates that the combined PD-observation undertakings were mutually beneficial toward 
the teacher-on-student effects desired and sought. 

This study indicates that use of a teacher observation system can reveal areas for improvement for which 
online, on-demand PD can provide coordinated remediation. This study further indicates, therefore, that the 
observational component of teacher evaluations should best be aligned with purposeful professional development. 
While teachers had open access to PD and were not limited to PD use only as directed by observers, it still remains 
compelling that teachers with more observations were also those with great PD use and whose students experienced 
greater collective improvements. 

Whether the improvements in teacher efficacy in student performance were attributable to the observations 
or to the PD is indiscernible. Past studies in teacher observations lacked a connectivity with PD as a potentially 
enhancing coordination of effort and investment. Past studies have verified the impact of online, on-demand PD for 
student gains (c.f. Shaha & Ellsworth, 2013, 2014), but teacher observations were not included in the design, so their 
coordinated impact is unknown. The combination of these research lackings makes this study more important and 
likely impactful but also sets a bar for next generation studies regarding programmatic approaches to improving 
teacher impacts on student performance. 

These findings reflect massive numbers of teachers and students within enormous numbers of schools, 
districts and U.S. states, thus they have far-reaching implications for practice and policy. As policymakers work to 
improve the education in their schools and in those throughout America and the broader educational world, the 
central focus of validation and verification efforts must include the evaluation of teachers for the purpose of 
improving their impact on students. Although observation-based teacher evaluations might be criticized and 
disconnected from the needs of students, this study shows that a coordinated approach involving PD 
recommendations and executions is impactful for student advances. This study shows that a well-designed 
observation approach, combined with a proven online, on-demand PD, provides a rigorous, differentiated link for 
enabling teachers to have their greatest impact on students. 
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